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Abstract 



The focus of this report is the implementation of Total Quality Management 
in ten DoD organizations. The participating organizations were all identified by the 
Federal Quality Institute as either winners or finalists for the Productivity/Quality 
Improvement Prototype (QIP) award sponsored by DoD and the Office of 
Management and Budget. Qualitative data collected included interviews with either 
top executives or TQM coordinators, documentation of quality management activities. 
A questionnaire survey was also administered to the executive steering committee of 
each organization providing a self-assessment of eight dimensions of quality 
management practices. The report describes the lessons learned, promising practices 
and the results of the self-assessment survey for the participating organizations. 
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RESULTS OF SELF-RATINGS ON 8 FACTORS OF QUALITY HANA6EMENT 
AT THE NAVAL AVIONICS CENTER 



OVERVIEW 

This report is derived from Master's thesis research conducted by Lt. 
Carolyn Applegate at the Naval Postgraduate School (Applegate, 1991). The focus 
of that thesis was to document the practices and experiences of ten exemplary DoD 
organizations in the implementation of Total Quality Management. Three types of 
data were collected from each organization. An interview was conducted with 
either the top executive (military or civilian) or TQM coordinator; documents 
describing TQM planning, achievements, activities were collected; and a 
questionnaire survey was distributed to the executive steering committee at each 
location to evaluate self-assessments of specific quality management practices. 
The thesis and a subsequent technical report (Applegate, Hocevar & Thomas, 1991) 
describe the results of both qualitative and quantitative data analyses 
presenting both lessons learned and promising practices for the entire sample of 
organizations. 

The focus of this companion report is on the results of the quantitative 
survey data gathered from one of the participating organizations -- the Naval 
Avionics Center (NAC). As described in the larger study, self-assessment of 
management practices can be a valuable tool in the achievement of quality 
improvements. The intent of this report is to provide NAC with data representing 
the top management group's self-assessment of the organization's quality 
management practices for their use in planning their continuous improvement 
strategies. 
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BACKGROUND 

For the past few years, top executives in a number of industries have been 
rethinking how to measure quality performance. During the 1980s, many managers 
involved in the quality movement came to realize that quality is a strategic 
weapon in a competitive world; this resulted in new performance measures such as 
tracking defect rates and response times (Troxell, 1981). The impetus of growth 
of the Total Quality concept, development of the Malcolm Baldridge National 
Quality Award, and increasingly stringent manufacturer demands on quality of 
supplier goods have led to a broadening of performance measures through an 
emphasis on quality. (Eccles, 1991) 

One problem with these new performance measurements is that relying on 
measurements of customer satisfaction, quality, and innovation is not as well 
ingrained in today's managers as financial performance measures. Current 
information resources do not readily support real-time management using new 
quality measures because they were designed based on traditional accounting 
systems. Real-time, operational measures of quality management, which broaden 
the basis of organizational performance measurement, can aid decision-makers to 
influence critical areas such as process management in order to improve 
performance. (Goldratt and Cox, 1986) 

Most organizations which use statistical process control tools collect 
performance data such as rework or defect rates that focus on production. 
However, these measures are limited in that they do not reflect organization-wide 
quality management. Various authors on quality recommend principles for 
effectively managing quality. These include Deming (1982;1986), Juran's (1986) 
quality trilogy, Crosby's (1979) zero-defect improvement programs, Ishikawa's 
(1985) total quality control, and Leonard and Sasser's (1982) identification of 
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quality levers. It is notable that all of these authors discuss the ideals of 
top management commitment, education, continuous improvement, and employee 
involvement. Examination of these and other principles provides a foundation for 
recognizing areas critical to any change in quality focus. 

The first thorough and systematic attempt to synthesize some of these 
quality concepts is shown in Table 1, adapted from a previous study (Saraph, 
Benson and Schroeder, 1989). Building on the writings of quality management 
authors, Saraph et al . propose organizational requirements for effective quality 
management. These organizational requirements are classified into eight critical 
factors necessary to achieve a successful shift to a quality focus. They 
include; the role of management, leadership and quality policy; the role of the 
quality department; training; product and service design; supplier quality 
management; process management; quality data and reporting; and employee 
relations. Measures of these critical factors can form a profile of an 
organization's quality management practices that provides a benchmark for making 
decisions to achieve higher or more ideal levels of quality within an 
organization. The eight critical factors and an explanation of what they 
represent are shown in Table 2. 

METHODOLOGY 

Choice of Organizations 

The DOD/Office of Management and Budget's (0MB) Productivity/Quality 
Improvement Prototype (QIP) was selected as the criterion for research 
participation. The purpose of this award is to recognize early successes, 
provide models for productivity improvement in other agencies, and provide 
visibility for high achievers. The Federal Quality Institute was contacted in 
order to develop a list of QIP winners and finalists since the award's inception 
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Table 1: ORGANIZATIONAL REQUIREMENTS FOR EFFECTIVE QUALITY MANAGEMENT EMPHASIZED BY 
SELECTED AUTHORS (adapted from Saraph et al., 1989) 

Critical (Oeming, 1982; (Juran, 1978; (Leonard and 

(Croaby, 1979) 1986) (Garvin, 1983) 1986; 1989) (Ishikawa, 1985) Sasser. 1982) 
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ensuring immediate 
action on quality 
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Table 2: CRITICAL FACTORS OF QUALITY MANAGEMENT 
(adapted from Saraph et al.,1989) 



Critical Factors 

of Quality Management Explanation of Critical Factors 



1. Role of 
management 
leadership and 
quality policy 


Acceptance of quality responsibility by top 
management and department heads. Evaluation of 
top management on quality. Participation by top 
management in quality improvement efforts. 
Specificity of quality goals. Importance 
attached to quality in relation to cost and 
schedule. Comprehensive quality planning. 


2. Role of the 
quality 
department 


Visibility and autonomy of the quality 
department. Quality department's access to top 
management. Use of quality staff for 
consultation. Coordination between quality 
department and other departments. Effectiveness 
of the quality department. 


3. Training 


Provision of statistical training, trade 
training, and quality-related training for all 
employees. 


4. Product /service 
design 


Thorough scrub-down process. Involvement of all 
affected departments in design reviews. 

Emphasis on producibility . Clarity of 
specifications. Emphasis on quality, not roll- 
out schedule. Avoidance of frequent redesigns. 


5. Supplier quality 
management 


Fewer dependable suppliers. Reliance on 
supplier process control. Strong 
interdependence of supplier and customer. 
Purchasing policy emphasizing quality rather 
than price. Supplier quality control. Supplier 
assistance in product development. 


6. Process 

management 


Clarity of process ownership, boundaries, and 
steps. Less reliance on inspection. Use of 
statistical process control. Selective 
automation. Fool-proof process design. 
Preventive maintenance. Employee self- 
inspection. Automated testing. 


7. Quality data and 
reporting 


Use of quality cost data. Feedback of quality 
data to employees and managers for problem 
solving. Timely quality measurement. Evaluation 
of managers and employees based on quality 
performance. Availability of quality data. 


8. Employee 
relations 


Implementation of employee involvement and 
quality circles. Open employee participation in 
quality decisions. Responsibility of employees 
for quality. Employee recognition for superior 
quality performance. Effectiveness of 
supervision in handling quality issues. On- 
going quality awareness of all employees. 
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in 1988. The resulting list identified 23 organizations, 11 of which were 
within DOD. All the DOD organizations were contacted and 10 agreed to 
participate. Each DOD organization provided a point of contact 
responsible for all administration concerned with this study. A list of 
the participating organizations is given in Appendix A. 

Questionnaire Survey 

A survey was adapted from a private sector study which 
developed and validated an instrument to measure the critical factors of 
quality management (Saraph, Benson and Schroeder, 1989). As described in 
the background section, the original survey developers based the questions 
on the current literature on quality management including such authors as 
Doming, Juran, Ishikawa, and Crosby. The questionnaire was validated 
using a sample from private sector organizations in Minneapolis, Minnesota 
including 3M, Control Data, and Northwest Airlines. 

The adapted form of the survey contains 66 questions composing the 
eight critical factors that assess a manager's perception of actual 
quality practices within his/her organization. Question wording was 
minimally modified to fit DoD organizations (e.g., "top executive" was 
changed to "commanding officer or executive director"). The modified 
survey as it was administered for this study is shown in Appendix B. 
Additional information on the reliability and validity of the survey 
instrument can be found in Appendix C. 

A typical survey item, as shown below, allows managers to indicate 
their perception of the degree or extent of a given practice within their 
organization: 
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Extent or Degree of Current Practice Is 
Very low Low Medium High Very High 
amount of final inspection, 1234 5 

review or checking 

Survey respondents were instructed to circle the number that represented 
their perception of quality management practices in their organization. 
Each critical factor was assessed using several component questions. For 
each component question and for each critical factor, the actual level of 
practice within or across organizations is represented by the average of 
the respondents' ratings for the component question or critical factor. 
The scale scores were calculated by summing the component item ratings and 
dividing by the number of items. The items comprising each critical 
factor along with the coefficient alpha statistic of internal consistency 
reliability are presented in Table 3. 

Survey Administration 

The survey respondents chosen within the ten organizations 
were members of each organization's quality council or executive steering 
committee, because these people serve to lead the quality focus within 
each organization. Survey data was collected in May, 1991. Each survey 
respondent assessed the degree or extent of actual quality management 
practices in his/her organization according to the measure described 
above. Table 4 lists the ten organizations anonymously, along with the 
number of responses anticipated and the number of survey responses 
received. At NAC (organization #2 in Table 4), eleven members of the 
Continuous Improvement Council participated in the study. 

Data Analysis 

Data were analyzed both in terms of individual items and the 
composite scales representing each of the eight critical factors of 
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Table 3; INTERNAL CONSISTENCY RELIABILITY ANALYSIS RESULTS 
FOR THE CRITICAL FACTORS OF QUALITY MANAGEMENT 
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Table 4: SURVEY RESPONSE 


Organization 


Surveys 

received 


Surveys 

sent 


Response 

rate 


#1 


7 


7 


100% 


#2 


11 


13 


85% 


#3 


14 


25 


56% 


#4 


10 


15 


67% 


#5 


10 


12 


83% 


#6 


11 


15 


73% 


#7 


20 


27 


74% 


#8 


8 


8 


100% 


#9 


6 


12 


50% 


#10 


5 


8 


63% 


Total 


102 


142 


72% 



quality management. Scale scores were calculated by summing the items 
comprising each scale and dividing by the number of items. This allows 
the scale scores to retain the same units of measure for ease of 
comparability. 

Two sets of analyses were performed on these data. The first 
analysis addressed NAC data solely and provides for all questions in the 
survey both the mean rating and the frequency of responses in each rating 
category. The mean rating represents the overall attitude of the group on 
the degree of practice for each particular aspect of quality management. 
The frequency demonstrates the degree of agreement that exists within the 
group on the status of managerial practice for each item. 

The second analysis compared the self-ratings of NAC with those from 
the remaining DoD organizations for all of the individual items comprising 
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the eight critical factors of quality management. This comparison uses a 
t-test comparison of means and a critical t-value appropriate for a 
significance level of .10. When the variances of the two groups (NAC and 
other DoD) were not statistically different, the pooled variance estimate 
for the t-value and significance level was used. If the variances were 
statistically different, the separate variance estimate was used. 

RESULTS EXAMINING NAC DATA ONLY 

Tables 5-12 present the mean or average rating of NAC participants 
for each item in the eight critical factors of quality management. Also 
shown are the frequency of responses for each rating category. The items 
are organized according to the 8 factors. The presentation of items is 
from highest to lowest rated within each factor category. In interpreting 
these means, note that a value of "3" represents a moderate rating of the 
extent of current practice. It is important to note that the ratings 
reflect the executive steering committee's self-evaluation of current TQM 
practices. In other words, the data reflect how the top management at NAC 
think they are doing on specific TQM-related activities. There are no 
norms for evaluating "good" or "poor" performance. The value of these 
data is in guiding discussion and critical analysis of the implications 
for action planning. 

Factor #1; Role of Top Management Leadership and Quality Policy 
The means and frequency of responses for factor #1 are 
presented in Table 5. Within this factor, "Role of top management 
leadership and quality policy," the most highly rated items are "the 
extent to which top management supports long-term quality improvement 
process," "the degree of participating by major branch or department heads 
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Factor #1; 



Rating 
5 4 3 2 1 

5 6 0 0 0 

2 7 2 0 0 
5 3 2 0 1 

3 5 12 0 
3 5 111 
14 5 10 
0 7 2 1 1 
2 3 3 2 1 
117 11 
0 5 2 3 1 

0 4 3 3 1 
0 17 3 0 

0 17 3 0 



Table 5 

Role of Top Management Leadership and Quality Policy 
Frequencies and Means of MAC Self-Ratings 



Mean 



Item 



4.5 extent to which top management supports long-term 
quality improvement process 

4.0 degree of participation by major branch or 

department heads in the quality improvement process 

4.0 degree to which top management considers quality 
management as a way to increase revenues/reduce 
costs 

3.8 extent to which the top executive assumes 

responsibility for quality performance 

3.7 importance attached to quality by top management in 
relation to cost and schedule objectives 

3.5 acceptance of responsibility for quality by major 
branch/department heads within the organization 

3.4 extent to which top management has objectives for 
quality performance 

3.3 amount of review of quality issues in top management 
meetings 

3.0 degree of comprehensiveness of the quality plan 

within the organization 

3.0 degree to which top management (commanding 

officer/executive director/major department heads) 
is evaluated for quality performance 

2.9 specificity of qual ity goals within the organization 

2.8 extent to which quality goals and policy are 

understood within the organization 

2.8 comprehensiveness of the goal -setting process for 
quality within the organization 

3.4 Overall Score for Critical Factor # 1 
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in the quality improvement process," and "the degree to which top 
management considers quality management as a way to increase 
revenues/reduce costs." Each of these three items has a mean rating of 
greater than 4.0. The aspects of factor #1 that receive the lowest self- 
rating all relate to quality goals. The specificity of quality goals, the 
extent to which these goals are understood within the organization, and 
the comprehensiveness of the goal-setting process for quality all receive 
ratings below 3.0. 

Factor #2: Role of the Quality Department 

All five items comprising this section (see Table 6) have mean 
ratings of at least 3.5, indicating a positive view of the managerial 
practices related to the quality department. The highest rating is for 
the quality department's access to top management. The lowest ratings in 
this section are for the autonomy of the department (x=3.6) and the 
effectiveness of the department in improving quality (x=3.5). 

Factor #3: Training 

Two items in the training factor are rated the highest in this 
category (see Table 7) with means greater than 4.0. These are "commitment 
of top management to employee training" and "training in the total quality 
concept. . .throughout the organization." The two items with the lowest 
self-ratings pertain to qual ity- related training given to non-supervisory 
employees, and training in advanced statistical techniques with means of 
3.2 and 2.7, respectively. 

Factor #4: Product/Service Design 

The range of means within this factor of quality management is 
fairly narrow (see Table 8). The highest self-rating is for coordination 
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Table 6 

Factor #2: Role of Quality Department 

Frequencies and Means of MAC Self-Ratings 



Rating 
5 4 3 2 1 


Mean 


Item 




8 3 0 0 0 


4.7 


quality department's access to top management 


3 5 3 0 0 


4.0 


visibility of the quality department 




2 5 4 0 0 


3.8 


amount of coordination between the 


qual ity 






department and other departments 




2 6 111 


3.6 


autonomy of the quality department 




0 6 4 1 0 


3.5 


effectiveness of the quality department in 


improving 






quality 






3.9 


Overall Score for Critical Factor # 2 
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Table 7 

Factor #3; Training 

Frequencies and Means of MAC Self-Ratings 



Rating 
5 4 3 2 1 

4 7 0 0 0 
4 6 10 0 

3 4 3 1 0 
3 3 4 1 0 

2 4 5 0 0 

1 4 6 0 0 

0 3 7 1 0 
0 3 3 4 1 



Mean 



Item 



4.4 commitment of the top management to employee 

training 

4.3 training in the "total quality concept" {i.e. 

philosophy of organization-wide responsibility for 
quality) throughout the organization 

3.8 availability of resources for employee training 

3.7 quality-related training given to managers and 

supervisors throughout the organization 

3.7 training in the basic statistical techniques (such 
as histograms and control charts) in the 
organization as a whole 

3.5 specific work-skills training (technical and 
vocational) given to non-supervisory employees 
throughout the organization 

3.2 quality-related training given to non-supervisory 
employees throughout the organization 

2.7 training in advanced statistical techniques (such as 
design of experiments and regression analysis) in 
the organization as a whole 

3.7 Overall Score for Critical Factor # 3 
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Table 8 

Factor #4: Product/Service Design 

Frequencies and Means of MAC Self-Ratings 



Rating 
5 4 3 2 1 

14 5 10 

0 6 4 1 0 

1 3 5 2 0 
0 1 10 0 0 
0 2 7 2 0 
0 18 2 0 



Mean 



Item 



3.5 coordination among affected departments in the 
process/service development process 

3.5 quality of new processes/services emphasized in 

relation to cost or schedule objectives 

3.3 extent to which implementation/producibil ity is 

considered in the process/service design process 

3.1 clarity of process/service specifications and 
procedures 

3.0 thoroughness of new process/service design reviews 
before the process/service is implemented/produced 

2.9 quality emphasis by customer service employees 

3.2 Overall Score for Critical Factor # 4 
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among departments in the development process (x=3.5); while the lowest is 
for the quality emphasis by customer service employees (x=2.9). As with 
the interpretation of all results, the implications of these ratings 
depends on the goals and capabilities of the organization. Within this 
factor, all ratings are clustered fairly closely to the midpoint of the 
rating scale, indicating that the executive group perceives a moderate 
degree of practice of these specific management activities in support of 
quality. Whether this represents a satisfactory achievement of quality 
practices in this area or becomes a benchmark for increased attention to 
these activities depends on the organizations's strategies for improving 
quality. 

Factor #5: Supplier Quality Management 

Of the eight items comprising this critical factor, only three 
show means greater than or equal to the midpoint of 3.0 (see Table 9). 
These include: "extent to which suppliers are selected based on quality 
rather than price or schedule" (x=3.5), "thoroughness of the supplier 
rating system" (x=3.2), and "clarity of specifications provided to 
suppliers" (x=3.0). The item with the lowest mean rating of 1.8 pertains 
to the involvement of the supplier in the product development process. 

Factor #6: Process Management 

The items that receive the highest rating in this factor 
category (see Table 10) all relate to aspects of inspection (incoming, 
acceptance sampling, final, and in-process). The means for these four 
items range from 3.4 to 3.7 and indicate that the practice of inspections 
is perceived to be above the "moderate" level by the executive group at 
MAC. The issue of the role of inspections within the philosophy of TQM, 
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Rating 
5 4 3 2 1 

15 4 10 

2 15 3 0 
0 2 7 2 0 
0 2 6 3 0 
0 2 3 4 1 

0 2 4 3 2 
0 2 3 5 1 

0 0 2 4 4 



Table 9 

Factor #5; Supplier Quality Management 
Frequencies and Means of MAC Self-Ratings 

Mean Item 



3.5 extent to which suppliers are selected based on 

quality rather than price or schedule 

3.2 thoroughness of the supplier rating system 

3.0 clarity of specifications provided to suppliers 

2.9 technical assistance provided to suppliers 

2.6 amount of education of suppliers by the 

organization 

2.5 reliance on reasonably few dependable suppliers 

2.5 extent to which longer term relationships are 

offered to suppliers 

1.8 involvement of the supplier in the product 

development process 

2.7 Overall Score for Critical Factor # 5 
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Table 10 

Factor #6: Process Management 

Frequencies and Means of MAC Self-Ratings 



Rating 
5 4 3 2 1 

1 5 4 0 0 

2 5 3 0 1 
1 6 3 0 1 
0 5 5 1 0 

0 4 5 2 0 
0 2 7 1 1 
0 2 5 4 0 
0 0 8 3 0 

0 17 2 1 

0 0 4 7 0 



Mean Item 

3.7 use of acceptance sampl ing to accept/reject lots or 
batches of work 

3.6 amount of incoming inspection, review or checking 

3.5 amount of final inspection, review or checking 

3.4 amount of in-process inspection, review or 
checking 

3.2 amount of preventive equipment maintenance 

2.9 degree of automation of the process 

2.8 stability of production schedule/work distribution 

2.7 clarity of work or process instructions given to 
employees 

2.7 extent to which inspection, review or checking of 
work is automated 

2.4 extent to which the design is "fool-proof" and 
minimizes chances of employee errors 

3.1 Overall Score for Critical Factor # 6 
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particularly as applied by DoD organizations is hotly debated. Therefore, 
it is difficult to interpret these numbers. If NAC has determined a 
certain level of inspection is critical to operations, the ratings on 
these items may reflect the achievement of organizational objectives in 
this area. If, however, the organization is attempting to decrease its 
reliance on inspections, the ratings may reflect an area needing increased 
attention. 

Three of the items in this factor category that have ratings below 
3.0 relate to process activities that can diminish the reliance on 
inspections for quality. These include the stability of production 
schedule and work distribution (x=2.8), the clarity of work or process 
instructions given to employees (x=2.7), and the extent to which the 
design is "fool-proof" and minimizes chances of employee errors (x=2.4). 

Factor #7: Quality Data and Reporting 

The ratings within this factor category (see Table 11) support 
the findings of the larger study from which these data are drawn 
(Applegate, 1991; Applegate, Hocevar & Thomas, 1991). That study reports 
the area of quality data is one of the most challenging being faced by all 
of the exemplary DoD organizations who participated. The self-rating data 
for NAC show that only two of the eight items have means above the 
midpoint -- availability of quality data and timeliness of quality data 
(3.2 and 3.0, respectively). The remaining items have means ranging from 
2.5 to 2.9. These ratings demonstrate that the extent to which quality 
data is used for quality management, is available to or displayed for non- 
supervisory employees, or is used to evaluate managerial performance are 
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Rating 
5 4 3 2 1 

1 2 6 2 0 

0 3 5 3 0 
0 2 7 1 1 

0 3 3 5 0 

0 15 5 0 
0 2 5 2 2 
0 2 4 3 2 
0 14 5 1 



Table 11 

Factor #7; Quality Data and Reporting 
Frequencies and Means of MAC Self-Ratings 



Mean 



Item 



3.2 availability of quality data (error rates, defect 
rates, scrap, defects) 

3.0 timeliness of the quality data 

2.9 extent to which quality data are available to 

managers and supervisors 

2.8 extent to which quality data (cost of quality, 

defects, errors, scrap, etc.) are used as tools to 
manage quality 

2.6 extent to which quality data are available to non- 
supervisory employees 

2.6 extent to which quality data, control charts, etc., 
are displayed at employees' workstations 

2.5 extent to which quality data are used to evaluate 
supervisor and managerial performance 

2.5 availability of cost of quality data in the 

organization 

2.8 Overall Score for Critical Factor # 7 
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practices that NAC may want to discuss as possible actions for improving 
the effective use of quality data. 

Factor #8; Employee Relations 

The final quality management category shows a range of self- 
ratings from 2.5 to 3.5 (see Table 12). The extent to which quality 
awareness building is ongoing among employees and the extent to which 
employee involvement type programs are implemented both have mean ratings 
of 3.5. The lowest mean rating (2.5) is for the extent to which employees 
are held responsible for error-free output. The amount of feedback 
provided employees on their quality performance and the degree of 
participation in quality decisions by non-supervisory employees both 
receive mean ratings of 2.7. The value of these results is that they 
focus on specific aspects of employee involvement in the quality 
improvement process. Thus, these practices can be discussed explicitly by 
the executive group as to their potential value and the best mechanisms 
for implementing them if such a decision should be made. 

RESULTS COMPARING NAC WITH OTHER DOD EXEMPLARY ORGANIZATIONS 

The preparation and presentation of results comparing NAC's self- 
evaluation with the self-evaluation of the other exemplary DoD 
organizations was done at the request of NAC's executive steering group. 
There are several cautions that should be mentioned before describing 
these results. First, these data do not represent objective performance 
ratings, but the self-evaluation of organizational members. For this 
reason, the interpretation of comparative results is difficult. It is 
possible, for example, that within the DoD sample there may be inflated 
self-perceptions as a result of having recently won the quality award -- 
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Table 12 

Factor #8; Employee Relations 
Frequencies and Means of MAC Self-Ratings 



Rating 
5 4 3 2 1 

0 6 4 1 0 
14 5 10 

1 2 5 2 1 
0 2 8 0 1 
0 3 5 3 0 
0 17 2 1 
0 0 8 3 0 
0 0 7 2 2 



Mean 



Item 



3.5 extent to which quality awareness building among 
employees Is ongoing 

3.5 extent to which quality circle or employee 
involvement type programs are implemented in the 
organization 

3.0 extent to which employees are recognized for 

superior quality performance 

3.0 effectiveness of supervisors in solving 
problems/issues 

3.0 effectiveness of the quality circle or employee 
Involvement type programs in the organization 

2.7 amount of feedback provided to employees on their 
quality performance 

2.7 degree of participation in quality decisions by 

non-supervisory employees 

2.5 extent to which employees are held responsible for 
error-free output 

3.0 Overall Score for Critical Factor # 8 
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"We just 'won'! We must be doing great!" This phenomenon is referred to 
as retrospective rationality (Pfeffer, 1982) and can lead to exaggerated 
differences between the self-ratings of the larger group that includes 
some award winners, and the self-ratings of NAC that may be casting a 
critical eye on its own performance in trying to enhance quality 
operations. However, the influence of retrospective rationality is only 
speculative; there is no way to evaluate the "accuracy" of the self-report 
data within the design constraints of this study. 

A second caution follows the warning of TQM advocates against 
seeking a "recipe for success" by following the model of another 
organization. To the extent that the achievement of other organizations 
can inspire NAC to further pursue aspects of quality management, these 
comparative results may prove to be motivational. There are potential 
benefits to the competition inherent in comparison data; but competition 
can also be dysfunctional if the pursuit of a certain activity is 
undertaken because others are rating themselves higher in that area rather 
than because it is determined to be the appropriate activity given the 
organization's own analyses of quality processes. 

In addition to the increased motivation mentioned above, another 
potential value of the comparative data is in highlighting areas where 
other organizations may have developed innovative practices. Such 
innovative practices are briefly described and contact points are 
identified in the full reports of this research project (see Applegate, 
1991; or Applegate, Hocevar & Thomas, 1991). A final value of the 
comparative results is in identifying areas where the challenge of 
implementing quality management is experienced by other exemplary 
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organizations. In other words, practices where NAC's self-rating is low 
may also be reported as low by other organizations, indicating the 
difficulty inherent in the changes required by continuous improvement. 
This occurs in a number of items where there is no significant difference 
between NAC's self-rating and the self-ratings of the other DoD 
organizations on items identified as problematic in the review of 
previously described results. 

Overview of Comparative Findings 

The t-test comparison of means for all 66 items was conducted 
with NAC (N=ll) and the remaining DoD organizational participants (N=91) 
as the two comparison groups. The two groups were found to have self- 
ratings that were n^ significantly different for more than 70% of the 
items. This indicates that in most of the areas of quality management 
practices, NAC's self-rating of the extent of current practice is the same 
as the self-rating of other exemplary DoD organizations. 

Practices Where NAC Ratings Were Higher Than DoD Ratings 

The practices where NAC's self-ratings were significantly 
higher than the DoD comparison group are presented in Table 13; the items 
are all from two categories of quality management -- supplier quality 
management and process management. In the first category, the extent to 
which suppliers are selected on the basis of quality rather than price or 
schedule and the thoroughness of the supplier rating system show NAC with 
mean ratings greater than the DoD comparison group. This result is not 
surprising given the Blue Ribbon Contractor program that has been 
developed at NAC and is documented in the larger report as an exemplary 
practice (see Applegate, 1991; Applegate, Hocevar & Thomas, 1991). 
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Table 13 



Quality Management Items Showing 
MAC Self-Ratings as Higher^ than 
Other DoD Benchmark Organizations 



Mean Ratings 

NAC DoD 

(N=ll) (N=91) 


Item 


5. 


Supplier quality management 


3.5 


2.2 


extent to which suppliers are selected based on quality 
rather than price or schedule 


3.2 


2.4 


thoroughness of the supplier rating system 


6. 


Process management 


3.7 


2.7 


use of acceptance sampling to accept/reject lots or 
batches of work 


3.6 


2.8 


amount of incoming inspection, review or checking 


3.5 


3.0 


amount of final inspection, review or checking 


2.7 


2.3 


extent to which inspection, review or checking of work is 
automated 



^ Differences were determined using a t-test comparison of 
mean ratings. All those reported are significant p<.10. All items 
not reported showed no difference between NAC and the other 
participating organizations. 
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The second category of process management has four items where 
NAC's mean ratings were significantly greater than the comparison group. 
These items all relate to inspections. As discussed in the description of 
results that examined solely NAC data, the interpretation of these 
findings is problematic. If the goal of the organization is to decrease 
reliance on inspections, this difference may indicate that NAC, while 
scoring higher, is not as far along as the other organizations perceive 
themselves to be in achieving this quality aim. However, it is also 
possible that NAC has determined that inspections cannot be decreased due 
to the nature of the process or product. The interpretation and 
determination of subsequent actions based on these results is 
fundamentally the responsibility of organizational members. 

Practices Where NAC Ratings Were Lower Than DoD Ratings 

The items showing NAC with mean self-ratings lower than the 
DoD comparison group are presented in Table 14. Several of the items 
included in these results have already been identified as potentially 
problematic in the discussion of results for the NAC data alone. For 
example, within the first category, the two items on the comprehensiveness 
of the goal setting process for quality and the extent of understanding 
within the organization of quality goals and policies were found to have 
the lowest mean self-ratings based on the examination of NAC data. Two 
additional items where other DoD organizations report higher means also 
relate to this theme: "degree of comprehensiveness of the quality plan 
within the organization," and "extent to which top management has 
objectives for quality performance." The final item in this category 
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Table 14 



Quality Management Items Showing 
MAC Self-Ratings as Lower^ than 
Other DoD Benchmark Organizations 



NAC 

(N=ll) 


DoD 

(N=91) 




1. Role of top 


management leadership and quality policy 


2.8 


3.3 


comprehensiveness of the goal -setting process for quality 
within the organization 


2.8 


3.3 


extent to which quality goals and policy are understood 
within the organization 


3.0 


3.5 


degree of comprehensiveness of the quality plan within the 
organization 


3.0 


3.5 


degree to which top management (commanding officer, 
executive director, major department heads) is evaluated 
for quality performance 


3.4 


3.9 


extent to which top management has objectives for quality 
performance 



3. Training 

3.2 3.8 quality-related training given to non-supervisory 

employees throughout the organization 



4. Product/service design 

2.9 3.6 quality emphasis by customer service employees 



^ Differences were determined using a t-test comparison of 
mean ratings. All those reported are significant p<.10. All items 
not reported showed no difference between NAC and the other 
participating organizations. 
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Table 14 (cont'd) 



NAC DoD 
(N=ll) (N=91) 

5. Suppl ier qual ity management 

1.8 2.3 involvement of the supplier in the product development 

process 



6. Process management 

2.7 3.2 clarity of work or process instructions given to employees 

7. Quality data and reporting 

2.9 3.4 extent to which quality data are available to managers 

and supervisors 



8. Employee relations 

3.0 3.6 effectiveness of the quality circle or employee 

involvement type programs in the organization 

3.0 3.5 effectiveness of supervisors in solving problems/issues 



3.0 3.5 extent to which employees are recognized for superior 

quality performance 



3.5 4.0 extent to which quality circle or employee involvement 

type programs are implemented in the organization 

2.7 3.0 degree of participation in quality decisions by non- 

supervisory employees 
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shows that the other DoD organizations report a greater use of quality 
performance criteria in the evaluation of top management. 

The next five categories of quality management each include only 
one item where NAC's self-rating was lower than the comparison group. In 
all cases, the items were identified as among the lowest self-rated in the 
previous discussion of NAC results. The items include: the extent of 
quality-related training given to non-supervisory employees, the quality 
emphasis by customer service employees, the involvement of the supplier in 
the product development process, and the clarity of work or process 
instructions given to employees. Again, these results suggest that the 
comparison data is redundant to the discussion of NAC data alone. 

The final category of quality management is employee relations. 
In this category there were four items that had lower ratings than the 
comparison group. Only one of these items was identified in the previous 
discussion of results -- "degree of participation in quality decisions by 
non-supervisory employees. Two of the remaining items deal with the 
extent and effectiveness of quality circle or employee involvement 
programs; the last item relates to the effectiveness of supervisors in 
solving problems/issues. 

DISCUSSION: HOW CAN THESE DATA BE USED? 

As described in the introduction to the presentation of results, 
there are no norms against which the data in this study can be compared. 
The primary value of the data is in the fact that they represent a self- 
assessment of the quality management practices at NAC as perceived by the 
top management group. The Applegate study (1991) found that nonfinancial 
quality measures of real-time management are a rare commodity. The data 
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reported here can serve two primary purposes. First, they can be used to 
guide discussion among the executive steering group regarding actions that 
might be planned to further enhance quality at NAC. In this way, the data 
can be viewed as input to the "check" phase of the PDCA cycle. The 
executive steering group has planned and implemented numerous continuous 
improvement initiatives within the organization and these data can be used 
as a form of assessment of progress based on the perceptions of the 
organizational leaders. The variance in perceptions may illuminate 
activities within one part of the organization that are not well known by 
other parts of the organization; or may identify further opportunities for 
improvement by virtue of an innovative idea of one of the steering group 
members . 

The second way in which these data may be used is as a baseline. 
As the organization continues its quality improvement activities, changes 
will occur. By using this survey with the same sample (or an expanded 
sample including Division managers or a other organizational members) over 
time, these changes can be tracked and the PDCA cycle continuously 
evaluated. This study is not intended to replace the need for more 
objective indicators of quality improvements; however, there is an 
important potential for insight into opportunities and barriers through 
the discussion of these data by the organizational decision makers. 
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APPENDIX A 



RESEARCH PARTICIPANTS 



The executive steering group or committee at each of the following 
organizations participated in the thesis survey. A point of contact (POC) 
is shown for each organization as well as the name and title for each 
organization's interviewee. 



Sacramento Air Logistics Center 
McClellan Air Force Base 
Sacramento, California 
Major General Michael D. Pavich, USAF 
Center Commander 
(POC Colonel Folz 
916-633-1164 
A/V 633-1164) 

Navy Aviation Supply Office 
(formerly Defense Industrial 
Supply Center) 

Philadelphia, Pennsylvania 
Rear Admiral James E. Eckleberger, USN 
Commanding Officer 
(POC Mr. Marvin Sandler 
215-697-1375 
A/V 442-1375) 

Naval Avionics Center 
Indianapolis, Indiana 
Captain Russell J. Henry, USN 
Commanding Officer 
(POC Mr. Thomas Sibert 
317-353-7470 
A/V 369-7470) 

Naval Aviation Depot 
Naval Station Norfolk 
Norfolk, Virginia 
Captain Thomas W. Hancock, USN 
Commanding Officer 
(POC Mr. Ross Haines 
804-445-1587) 

Norfolk Naval Shipyard 
Portsmouth, Virginia 
Captain James T. Taylor, USN 
Commanding Officer 
(POC Mr. Duff Porter 
804-396-7092) 
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Naval Ship Systems 
Engineering Station 
Philadelphia, Pennsylvania 
Captain Dennis K. Kruse, USN 
Commanding Officer 
(POC Mr. James Summers 
215-897-7828) 

1926th Communications-Computer Group 
Warner Robins Air Logistics Center 
Warner Robins Air Force Base, Georgia 
Mr. Clifford E. Carroll 
Executive Director 
(POC Ms. Jeanie Spence 
912-926-7687 
A/V 468-7687) 

Naval Supply Center 
San Diego, California 
Captain Gary D. Lynn, USN 
Executive Officer 
(POC Ms. Donna Tierney 
619-532-1689 
A/V 522-1689) 

Naval Aviation Depot 
Marine Corps Air Station 
Cherry Point, North Carolina 
Mr. John C. Adams 
TQM Coordinator 
(POC Mr. John Adams 
919-466-7403 
A/V 582-7403) 

Navy Aviation Supply Office, Code 10 
(formerly Naval Publications 
and Forms Center) 

Philadelphia, Pennsylvania 
Lieutenant Commander Kenneth K. Kittredge, USN 
Director, Publications and Forms 
(POC Mr. Dennis Cronin 
215-697-4919 
A/V 442-4919) 



34 



APPENDIX B: SURVEY QUESTIONS 



Extent or Degree of Current Practice Is 
Very Low Low Medium High Very High 



Extent to which the top I 2 

executive assumes responsibility 
for quality performance 

Visibility of the quality 1 2 

department 

Specific work-skills training 1 2 

(technical and vocational) 
given to non-supervisory employees 
throughout the organization 

Thoroughness of new process/ 1 2 

service design reviews before 
the process/service is 
implemented/produced 

Extent to which suppliers are 1 2 

selected based on quality 
rather than price or schedule 

Use of acceptance sampling to 1 2 

accept/reject lots or batches 
of work 

Availability of cost of quality 1 2 

data in the organization 

Extent to which quality circle 1 2 

or employee involvement type 
programs are implemented in 
the organization 

Acceptance of responsibility 1 2 

for quality by major branch/ 
department heads within the 
organization 

Quality department's access 1 2 

to top management 

1 2 



3 4 5 

3 4 5 

3 4 5 

3 4 5 

3 4 5 

3 4 5 

3 4 5 

3 4 5 

3 4 5 

3 4 5 

3 4 5 



Quality-related training given 
to non-supervisory employees 
throughout the organization 
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Extent or Degree of Current Practice Is 
Very Low Low Medium High Very High 



Coordination among affected 
departments in the process/ 
service development process 

Thoroughness of the 
supplier rating system 

Amount of preventive 
equipment maintenance 

Availability of quality data 
(error rates, defect rates, 
scrap, defects) 

Effectiveness of the quality 
circle or employee involvement 
type programs in the organization 

Degree to which top management 
(commanding officer/executive 
director/major department 
heads) is evaluated for quality 
performance 

Autonomy of the quality 
department 

Quality-related training 
given to managers and 
supervisors throughout the 
organization 

Quality of new processes/ 
services emphasized in relation 
to cost or schedule objectives 

Reliance on reasonably few 
dependable suppliers 

Extent to which inspection, 
review, or checking of work 
is automated 

Timeliness of the quality 
data 

Extent to which employees 
are held responsible for 
error-free output 



1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 
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Extent or Degree of Current Practice Is 
Very Low Low Medium High Very High 



Extent to which top management 123 

supports long-term quality 
improvement process 

Amount of coordination 123 

between the quality department 
and other departments 

Training in the "total 123 

quality concept"(i .e. philosophy 
of organization-wide 
responsibility for quality) 
throughout the organization 

Clarity of process/service 123 

specifications and procedures 

Amount of education of 123 

suppliers by the organization 

Amount of incoming inspection, 1 2 3 

review, or checking 

Extent to which quality data 123 

(cost of quality, defects, 
errors, scrap, etc.) are used 
as tools to manage quality 

Amount of feedback provided 123 

to employees on their quality 

performance 

Degree of participation by 123 

major branch/department heads 
in the quality improvement process 

Effectiveness of the quality 
department in improving quality 

Training in the basic 
statistical techniques (such 
as histograms and control 
charts) in the organization 
as a whole 

Extent to which implementation 
/producibil ity is considered 
in the process/service design 
process 



1 2 

1 2 3 

1 2 3 



4 5 

4 5 

4 5 



4 5 

4 5 

4 5 

4 5 



4 5 

4 5 

4 5 

4 5 



4 5 
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Extent or Degree of Current Practice Is 
Very Low Low Medium High Very High 



Technical assistance provided 
to suppliers 

Amount of in-process 
inspection, review, or checking 

Extent to which quality 
data are available to non- 
supervisory employees 

Degree of participation in 
quality decisions by non- 
supervisory employees 

Extent to which top 
management has objectives 
for quality performance 

Training in advanced 
statistical techniques (such 
as design of experiments and 
regression analysis) in the 
organization as a whole 

Quality emphasis by customer 
service employees 

Involvement of the supplier 
in the product development 
process 

Amount of final inspection, 
review, or checking 

Extent to which quality data 
are available to managers and 
supervisors 

Extent to which quality 
awareness building among 
employees is ongoing 



2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 



1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 
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Extent or Degree of Current Practice Is 
Very Low Low Medium High Very High 



Specificity of quality goals 
within the organization 

Commitment of the top 
management to employee 
training 

Extent to which longer term 
relationships are offered to 
suppliers 

Stability of production 
schedule/work distribution 

Extent to which quality data 
are used to evaluate supervisor 
and managerial performance 

Extent to which employees 
are recognized for superior 
quality performance 

Comprehensiveness of the 
goal-setting process for 
quality within the organization 

Availability of resources for 
employee training in the 
organization 

Clarity of specifications 
provided to suppliers 

Degree of automation of the 
process 

Extent to which quality data, 
control charts, etc., are 
displayed at employee’s 
work stations 



2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 



1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 
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Effectiveness of supervisors 
in solving problems/issues 

Extent to which quality goals 
and policy are understood 
within the organization 

Extent to which process design 
is "fool-proof and minimizes 
chances of employee errors 

Importance attached to quality 
by top management in relation 
to cost and schedule objectives 

Clarity of work or process 
instructions given to employees 

Amount of review of quality 
issues in top management meetings 

Degree to which top management 
considers quality management as 
a way to increase revenues/reduce 
costs . 

Degree of comprehensiveness 
of the quality plan within 
the organization 



Extent or Degree of Current Practice Is 
Very Low Low Medium High Very High 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 
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APPENDIX C: STATISTICAL ANALYSIS OF SURVEY RESULTS 



The survey used in this study is adapted from an instrument 
developed and validated by Saraph, Benson and Schroeder (1989). The 
Saraph et al . citation provides substantial evidence for the validity of 
the eight critical factors of quality management by evaluating content 
validity, criterion-related validity and construct validity. 

For this research, the survey was modified by dropping 12 questions 
that were determined to be unreliable in the original study. As stated in 
the body of this report, additional modifications to wording were made to 
fit DoD organizations. The modified survey, containing 66 questions, was 
formally reviewed by two civilian professors of management in order to 
ensure the language changes would ease comprehension of the survey 
questions by the targeted audience, without changing the substantive 
intent of the questions. 

The reliability of the survey data collected for this study was 
evaluated using the internal consistency method. Cronbach's alpha, which 
is well suited to attitude instruments in which multiple questions are 
used to address a single dimension (i.e. training, process management), 
was chosen to assess internal consistency reliability (Jaeger, 1983). The 
SPSS/PC+ reliability program was used to conduct the analysis (Norusis, 
1990). Missing data, which was minimal, was handled by substituting the 
median score for each survey question, so as not to exclude any survey 
responses from this study. 

Results for the eight critical factors' reliability are detailed in 
Table 3 (see the Methodology section), which shows that the reliability 
coefficients or alpha scores ranged from .73 to .91, all of which are 
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considered adequate for reliability of research instruments. This 
analysis demonstrates that different questions intended to measure the 
same critical factor show convergence (Cronbach, 1951; Jaeger, 1983; Yin, 
1984). These results further supported reliability evidence presented by 
the original developers of the instrument. 

A correlation matrix for the critical factors of quality management 
was completed as an additional measure of discriminant validity, and is 
detailed in the following table. Because the factors all deal with 
quality management, significant correlations are to be expected. All but 
four intercorrelations show at least 50% unique variance, thus supporting 
discriminant validity. The highest intercorrelation was found between 
leadership and employee relations (r=.79). This suggests that these two 
dimensions have 62% variance in common, and 38% unique variance. While 
this is not a strong indication of discriminant validity, it was felt to 
be sufficient for purposes of this study. 
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TABLE C: SCALE TO SCALE CORRELATION MATRIX FOR THE 
CRITICAL FACTORS OF QUALITY MANAGEMENT 


Critical Factor 1 


2 


Scale # 
3 4 


5 


6 


7 


8 


Role of management 1 .0 

leadership and quality 
policy 
(scale #1) 


.58 


.66 .71 


.31 


.46 


.72 


.79 


Role of the quality 
department 
(scale #2) 


1.0 


.49 .58 


.32 


.37 


.48 


.58 


Training 
(scale #3) 




1.0 .56 


.42 


.43 


.66 


.66 


Product /service design 
(scale #4) 




1.0 


.40 


.58 


.71 


.69 


Supplier quality 
management 
(scale #5) 






1.0 


.59 


.42 


.37 


Process management 
(scale #6) 








1.0 


.64 


.46 


Quality data and repotting 
(scale #7) 










1.0 


.74 


Employee relations 
(scale #8) 












1.0 



43 



REFERENCES 



Applegate, Carolyn, Total Quality Management in the Department of Defense: 
Lessons Learned. Innovative Practices, and Quality Measurements . 
Master of Science Thesis, Department of Administrative Sciences, 
Naval Postgraduate School, Monterey, CA, September, 1991. 

Applegate, Carolyn, Hocevar, Susan Page, and Thomas, Kenneth W., Total 
Quality Management in Ten Examolarv Department of Defense 
Organizations: Lessons Learned. Innovative Practices, and Quality 

Measurements . Technical Report (NPS-AS-003) , Department of 
Administrative Sciences, Naval Postgraduate School, Monterey, CA, 
November, 1991. 

Cronbach, J. L., "Coefficient Alpha and the Internal Structure of Tests," 
Psvchometrika . Vol . 16, No. 3, September 1951, pp. 297-334. 

Crosby, P.B., Quality is Free . New American Library, 1979. 

Deming, W.E., "Improvement of Quality and Productivity through Action by 
Management," National Productivity Review . Winter 1981-1982, pp. 12- 
22 . 

Deming, W.E., Out of the Crisis . Massachusetts Institute of Technology, 
Center for Advanced Engineering, 1986. 

Eccles, Robert G., "The Performance Measurement Manifesto," Harvard 
Business Review . January-February 1991, pp. 131-139. 

Garvin, David A., "Quality on the Line," Harvard Business Review . 
September-October 1983, pp. 64-75. 

Goldratt, Eliyahu M. and Cox, Jeff, Ihe Goal : A Process M Ongoing 
Improvement , revised edition. North Rivers Press, Inc., 1986. 

Ishikawa, Kaoru, What is Total Quality Control? the Japanese Wav . 
Prentice-Hall, Inc., 1985. 

Jaeger, Richard M. , Statistics : A Spectator Sport . Sage Publications, 
Inc., 1983, pp. 93-101. 

Juran, J.M., "Japanese and Western Quality: A Contrast in Methods and 
Results," Management Review . Vol. 67, No. 11, November 1978, pp. 26- 
29. 



Juran, J.M., "The Quality Trilogy." Quality Progress . Vol. 9, No. 8, 1986, 
pp. 19-24. 

Juran, J.M., Juran gji Leadership for Quality : ^ Executive Handbook . The 
Free Press, 1989. 



44 



Leonard, Frank S., and Sasser, W. Earl, "The Incline of Quality," Harvard 
Business Review . September-October 1982, pp. 163-171. 

Norusis, Marija J., $P$$/PC+ Statistics for the IBM PC/XT/AT and PS/2 . 
SPSS Inc., 1990. 

Pfeffer, Jeffrey, Organizations and organization theory , Boston: Pitman, 

1982. 

Saraph, Jayant V., Benson, P. George, and Schroeder, Roger G., "An 
Instrument for Measuring the Critical Factors of Quality 

Management," Decision Sciences . Vol . 20, 1989, pp. 810-829. 

Troxell, Joseph R., "Service Time Quality Standards," Quality Progress . 
Vol. XIV, No. 9, September 1981, pp. 27-33. 

Yin, Robert K., Case Study Research : Design and Methodology . Sage 
Publications, Inc., 1984. 



45 



DISTRIBUTION LIST 



Number of Copies 



Defense Technical Information Center 2 

Cameron Station 

Alexandria, Virginia 22304-6145 

National Technical Information Service 1 

5285 Port Royal Road 
Springfield, Virginia 22161 

Dudley Knox Library, Code 0142 2 

Naval Postgraduate School 
Monterey, California 93943-5002 

Office of Research Administration 1 



Code 08 

Naval Postgraduate School 
Monterey, CA 93943 

Library, Center for Naval Analyses 1 

4401 Ford Avenue 
Alexandria, VA 22302-0268 

Department of Administrative Sciences Library 1 

Code AS 

Naval Postgraduate School 
Monterey, CA 93943 



Dr. Kenneth W. Thomas, Code AS/TH 
Department of Administrative Sciences 
Naval Postgraduate School 
Monterey, California 93943-5000 

Dr. Susan Page Hocevar, Code AS/HC 
Department of Administrative Sciences 
Naval Postgraduate School 
Monterey, California 93943-5000 

David Fogelman 
Naval Avionics Center D/500 
6000 East 21st Street 
Indianapolis, Indiana 46219-2189 

Thomas Sibert 

Naval Avionics Center D/470 
6000 East 21st Street 
Indianapolis, Indiana 46219-2189 



2 



10 



5 



15 



Commander, Naval Base 
Norfolk, Virginia 23511-6002 
Attn: LT Carolyn L. Applegate 



1 



DUDLEY KNOX LIBRARY 



3 2768 00342566 



